[SPARK-55128][INFRA] Restore SQL tests by pin 'pandas==2.3.3' by zhengruifeng · Pull Request #53910 · apache/spark

zhengruifeng · 2026-01-22T03:45:23Z

What changes were proposed in this pull request?

Restore Restore SQL tests by pin 'pandas<3'

Why are the changes needed?

pandas 3 is just released, and fail sql tests

https://github.com/apache/spark/actions/runs/21232213791/job/61092886134

currently pandas 3 doesn't affect python tests too much:
1, in dev/requirements.txt, the latest mlflow==3.8.1 requires: pandas<3
2, pandas==2.3.3 is pinned in most places

Does this PR introduce any user-facing change?

no

How was this patch tested?

ci

Was this patch authored or co-authored using generative AI tooling?

no

github-actions · 2026-01-22T03:45:31Z

JIRA Issue Information

=== Test SPARK-55128 ===
Summary: Restore SQL tests by pin 'pandas<3'
Assignee: None
Status: Open
Affected: ["4.2"]

This comment was automatically generated by GitHub Actions

zhengruifeng · 2026-01-22T07:08:21Z

all sql tests passed
https://github.com/zhengruifeng/spark/actions/runs/21236806847/job/61106499975

LuciferYang · 2026-01-22T07:11:53Z

.github/workflows/build_and_test.yml

      if: (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-')) || contains(matrix.modules, 'connect') || contains(matrix.modules, 'yarn')
      run: |
-        python3.11 -m pip install 'numpy>=1.22' pyarrow pandas pyyaml scipy unittest-xml-reporting 'lxml==4.9.4' 'grpcio==1.76.0' 'grpcio-status==1.76.0' 'protobuf==6.33.0' 'zstandard==0.25.0'
+        python3.11 -m pip install 'numpy>=1.22' pyarrow 'pandas==2.3.3' pyyaml scipy unittest-xml-reporting 'lxml==4.9.4' 'grpcio==1.76.0' 'grpcio-status==1.76.0' 'protobuf==6.33.0' 'zstandard==0.25.0'


Is this the only place where the version needs to be pinned? For example, is it not necessary in other places like requirements.txt?

file requirements.txt is only for developer, we may want to dev/test with pandas 3.0 in local env,
and current mlflow requires pandas<3

### What changes were proposed in this pull request? Restore Restore SQL tests by pin 'pandas<3' ### Why are the changes needed? pandas 3 is just released, and fail sql tests https://github.com/apache/spark/actions/runs/21232213791/job/61092886134 currently pandas 3 doesn't affect python tests too much: 1, in `dev/requirements.txt`, the latest `mlflow==3.8.1` requires: `pandas<3` 2, `pandas==2.3.3` is pinned in most places ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? ci ### Was this patch authored or co-authored using generative AI tooling? no Closes #53910 from zhengruifeng/restore_sql. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org> (cherry picked from commit dafb2cd) Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

zhengruifeng · 2026-01-22T07:22:27Z

merged to master/4.1

…3' for maven daily test ### What changes were proposed in this pull request? Similar to #53910, this pr pins the pandas version to 2.3.3. ### Why are the changes needed? To restore SQL tests for maven daily test. - https://github.com/apache/spark/actions/runs/21249870076/job/61148348328 ``` - udf/postgreSQL/udf-case.sql - Scalar Pandas UDF *** FAILED *** udf/postgreSQL/udf-case.sql - Scalar Pandas UDF Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0 Expected Some("struct<Two:string,i:int,f:double,i:int,j:int>"), but got Some("struct<>") Schema did not match for query #30 SELECT '' AS `Two`, * FROM CASE_TBL a, CASE2_TBL b WHERE udf(COALESCE(f,b.i) = 2): -- !query SELECT '' AS `Two`, * FROM CASE_TBL a, CASE2_TBL b WHERE udf(COALESCE(f,b.i) = 2) -- !query schema struct<> -- !query output org.apache.spark.SparkRuntimeException { "errorClass" : "CAST_INVALID_INPUT", "sqlState" : "22018", "messageParameters" : { "ansiConfig" : "\"spark.sql.ansi.enabled\"", "expression" : "'nan'", "sourceType" : "\"STRING\"", "targetType" : "\"BOOLEAN\"" }, "queryContext" : [ { "objectType" : "", "objectName" : "", "startIndex" : 62, "stopIndex" : 85, "fragment" : "udf(COALESCE(f,b.i) = 2)" } ] } (SQLQueryTestSuite.scala:681) ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? monitor maven daily test after pr merged ### Was this patch authored or co-authored using generative AI tooling? No Closes #53933 from LuciferYang/SPARK-55128-FOLLOWUP. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

…3' for maven daily test ### What changes were proposed in this pull request? Similar to #53910, this pr pins the pandas version to 2.3.3. ### Why are the changes needed? To restore SQL tests for maven daily test. - https://github.com/apache/spark/actions/runs/21249870076/job/61148348328 ``` - udf/postgreSQL/udf-case.sql - Scalar Pandas UDF *** FAILED *** udf/postgreSQL/udf-case.sql - Scalar Pandas UDF Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0 Expected Some("struct<Two:string,i:int,f:double,i:int,j:int>"), but got Some("struct<>") Schema did not match for query #30 SELECT '' AS `Two`, * FROM CASE_TBL a, CASE2_TBL b WHERE udf(COALESCE(f,b.i) = 2): -- !query SELECT '' AS `Two`, * FROM CASE_TBL a, CASE2_TBL b WHERE udf(COALESCE(f,b.i) = 2) -- !query schema struct<> -- !query output org.apache.spark.SparkRuntimeException { "errorClass" : "CAST_INVALID_INPUT", "sqlState" : "22018", "messageParameters" : { "ansiConfig" : "\"spark.sql.ansi.enabled\"", "expression" : "'nan'", "sourceType" : "\"STRING\"", "targetType" : "\"BOOLEAN\"" }, "queryContext" : [ { "objectType" : "", "objectName" : "", "startIndex" : 62, "stopIndex" : 85, "fragment" : "udf(COALESCE(f,b.i) = 2)" } ] } (SQLQueryTestSuite.scala:681) ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? monitor maven daily test after pr merged ### Was this patch authored or co-authored using generative AI tooling? No Closes #53933 from LuciferYang/SPARK-55128-FOLLOWUP. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org> (cherry picked from commit 3f1c9a3) Signed-off-by: yangjie01 <yangjie01@baidu.com>

pan3793 · 2026-02-11T04:51:33Z

@zhengruifeng I guess this is also required for branch-4.0, it seems broken for a while

Restore Restore SQL tests by pin 'pandas<3' pandas 3 is just released, and fail sql tests https://github.com/apache/spark/actions/runs/21232213791/job/61092886134 currently pandas 3 doesn't affect python tests too much: 1, in `dev/requirements.txt`, the latest `mlflow==3.8.1` requires: `pandas<3` 2, `pandas==2.3.3` is pinned in most places no ci no Closes apache#53910 from zhengruifeng/restore_sql. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

…3' for maven daily test Similar to apache#53910, this pr pins the pandas version to 2.3.3. To restore SQL tests for maven daily test. - https://github.com/apache/spark/actions/runs/21249870076/job/61148348328 ``` - udf/postgreSQL/udf-case.sql - Scalar Pandas UDF *** FAILED *** udf/postgreSQL/udf-case.sql - Scalar Pandas UDF Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0 Expected Some("struct<Two:string,i:int,f:double,i:int,j:int>"), but got Some("struct<>") Schema did not match for query apache#30 SELECT '' AS `Two`, * FROM CASE_TBL a, CASE2_TBL b WHERE udf(COALESCE(f,b.i) = 2): -- !query SELECT '' AS `Two`, * FROM CASE_TBL a, CASE2_TBL b WHERE udf(COALESCE(f,b.i) = 2) -- !query schema struct<> -- !query output org.apache.spark.SparkRuntimeException { "errorClass" : "CAST_INVALID_INPUT", "sqlState" : "22018", "messageParameters" : { "ansiConfig" : "\"spark.sql.ansi.enabled\"", "expression" : "'nan'", "sourceType" : "\"STRING\"", "targetType" : "\"BOOLEAN\"" }, "queryContext" : [ { "objectType" : "", "objectName" : "", "startIndex" : 62, "stopIndex" : 85, "fragment" : "udf(COALESCE(f,b.i) = 2)" } ] } (SQLQueryTestSuite.scala:681) ``` No monitor maven daily test after pr merged No Closes apache#53933 from LuciferYang/SPARK-55128-FOLLOWUP. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

Backport #53910 and #53933 to branch-4.0 ### What changes were proposed in this pull request? Pandas 3.0 released, pin 'pandas==2.3.3' to recover the CI. ### Why are the changes needed? Recover CI. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Wait for GHA result. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #54263 from pan3793/SPARK-55128-4.0. Lead-authored-by: Ruifeng Zheng <ruifengz@apache.org> Co-authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

test

131b05b

github-actions bot added the INFRA label Jan 22, 2026

zhengruifeng changed the title ~~[SPARK-55128][INFRA] Restore Restore SQL tests by pin 'pandas<3'~~ [SPARK-55128][INFRA] Restore SQL tests by pin 'pandas<3' Jan 22, 2026

2.3.3

8af0db3

zhengruifeng changed the title ~~[SPARK-55128][INFRA] Restore SQL tests by pin 'pandas<3'~~ [SPARK-55128][INFRA] Restore SQL tests by pin 'pandas==2.3.3' Jan 22, 2026

zhengruifeng requested review from LuciferYang and cloud-fan January 22, 2026 07:08

LuciferYang reviewed Jan 22, 2026

View reviewed changes

LuciferYang approved these changes Jan 22, 2026

View reviewed changes

zhengruifeng closed this in dafb2cd Jan 22, 2026

zhengruifeng deleted the restore_sql branch January 22, 2026 07:22

LuciferYang mentioned this pull request Jan 23, 2026

[SPARK-55128][INFRA][FOLLOWUP] Restore SQL tests by pin 'pandas==2.3.3' for maven daily test #53933

Closed

pan3793 mentioned this pull request Feb 11, 2026

[SPARK-55128][INFRA][4.0] Restore SQL tests by pin 'pandas==2.3.3' #54263

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-55128][INFRA] Restore SQL tests by pin 'pandas==2.3.3'#53910

[SPARK-55128][INFRA] Restore SQL tests by pin 'pandas==2.3.3'#53910
zhengruifeng wants to merge 2 commits intoapache:masterfrom
zhengruifeng:restore_sql

zhengruifeng commented Jan 22, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 22, 2026

Uh oh!

zhengruifeng commented Jan 22, 2026

Uh oh!

LuciferYang Jan 22, 2026

Uh oh!

zhengruifeng Jan 22, 2026 •

edited

Loading

Uh oh!

zhengruifeng commented Jan 22, 2026

Uh oh!

pan3793 commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

zhengruifeng commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions bot commented Jan 22, 2026

JIRA Issue Information

Uh oh!

zhengruifeng commented Jan 22, 2026

Uh oh!

LuciferYang Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

zhengruifeng Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhengruifeng commented Jan 22, 2026

Uh oh!

pan3793 commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhengruifeng commented Jan 22, 2026 •

edited

Loading

zhengruifeng Jan 22, 2026 •

edited

Loading